1. Field of the Invention
The present invention relates to an image processing apparatus and method and, more particularly, to an image processing apparatus and method which correct a degraded image by using image recovery processing.
2. Description of the Related Art
Since the digitization of information has allowed images to be processed as signal values, there have been proposed various correction processing methods for sensed images. When an object is sensed and imaged by a digital camera, the obtained image suffers from a degree of degradation in quality. The degradation in image quality is caused particularly by the aberrations of the optical imaging system used to image the object.
The causes of blur components in an image include the spherical aberration, comatic aberration, field of curvature, and astigmatism of an optical system. Each of the blur components of an image due to these aberrations indicates that a light beam emerging from one point of an object is formed into an image with a spread, which should converge into one point on an imaging plane without any aberration or any influence of diffraction. This state is called a PSF (Point Spread Function) in optical terms but will be referred to herein as a blur component in image terms. Blur in an image may indicate a defocused image, but is used to indicate herein an image blurred due to the influences of the above aberrations of the optical system, even if it is in focus. In addition, color fringing on color images due to the chromatic aberration on the axis, spherical aberration of color, and comatic aberration of color of optical systems can be regarded as different ways of blurring at different wavelengths.
The OTF (Optical Transfer Function) obtained by Fourier transform of a PSF is frequency component information of an aberration, which is expressed by a complex number. The absolute value of an OTF, that is, an amplitude component, will be referred to as an MTF (Modulation Transfer Function), and a phase component will be referred to as a PTF (Phase Transfer Function). That is, an MTF and PTF are respectively the frequency characteristics of an amplitude component and phase component of an image's degradation due to aberrations. In this case, a phase component is represented as a phase angle byPTF=tan−1(Im(OTF)/Re(OTF))  (1)where Re(OTF) and Im(OTF) respectively represent the real part and the imaginary part of the OTF.
As described above, the OTF of an optical imaging system causes degradations in the amplitude component and phase component of an image. For this reason, a degraded image asymmetrically blurs at each point of the imaged object like a comatic aberration.
In addition, the chromatic aberration of magnification occurs when image forming positions shift due to the differences in image forming magnification at different wavelengths of light, and an image sensing apparatus acquires the shifts as R, G, and B color components in accordance with the spectral characteristics. Image spreading occurs due to image forming position shifts at different wavelengths within each color component, that is phase shifts, as well as image forming position shifts between R, G, and B components. To be precise, the chromatic aberration of magnification is not simple color fringing due to horizontal shifts. However, color fringing will be used herein as a synonym of the chromatic aberration of magnification.
As a method of correcting degradations in amplitude (MTF) and phase (PTF), a method of correcting them by using the information of the OTF of an optical imaging system is known. This method is called “image recovery” and “image restoration”. The processing of correcting degradation in image by using the information of the OTF of an optical imaging system will be referred to as image recovery processing.
The following is an outline of image recovery processing. Letting g(x, y) be a degraded image, f(x, y) be the original image, and h(x, y) be the PSF obtained by inverse Fourier transform of the optical transfer function, equation (2) given below holds:g(x,y)=h(x,y)*f(x,y)  (2)where * represents convolution and (x, y) represents coordinates on the image.
When this equation is converted into a display form on a frequency plane by Fourier transform, it becomes a form of product for each frequency as represented by equation (3):G(u,v)=H(u,v)·F(u,v)  (3)where H is the function obtained by Fourier transform of a PSF, and hence represents an OTF, and (u, v) represents coordinates on a two-dimensional frequency plane, that is, a frequency.
That is, in order to obtain the original image from the sensed degraded image, both sides of equation (3) may be divided by H as represented by equation (4) given below.G(u,v)/H(u,v)=F(u,v)  (4)Returning F(u, v) to the real plane by inverse Fourier transform can obtain the original image f(x, y) as a recovered image.
Letting R be the value obtained by inverse Fourier transform of equation (4), it is also possible to obtain the original image by performing convolution processing for an image on the real surface, as represented by equation (5):g(x,y)*R(x,y)=f(x,y)  (5)where R(x, y) is called an image recovery filter. An actual image, however, includes noise components. For this reason, using an image recovery filter generated by taking the perfect reciprocal of the OTF in the above manner will amplify noise components together with the degraded image. In general, therefore, a proper image cannot be obtained. In this respect, for example, there is known a method of suppressing the recovery ratio on the high-frequency side of an image in accordance with the intensity ratio between an image signal and a noise signal, such as a method using a Wiener filter. As a method of correcting degradation in the color fringing component of an image, for example, the degradation is corrected by correcting the above blur components so that the amount of blur is made uniform for the respective color components of the image.
In this case, since the OTF varies in accordance with image sensing conditions such as a zooming position and an aperture diameter, it is necessary to change the image recovery filter used for image recovery processing accordingly.
For example, Japanese Patent Laid-Open No. 2006-238032 discloses image recovery processing which is performed upon setting a minute spread in the PSF after image recovery. Japanese Patent No. 03532368 discloses a technique of eliminating an image blur in an endoscope for observing the interior of the living body by using a PSF corresponding to a fluorescence wavelength to be used with respect a range outside the in-focus range of an image sensing means. Since the fluorescence is weak, an object optical system with a small f-number is required. This leads to a decrease in focal depth. This technique is therefore designed to obtain an in-focus image by performing image recovery processing with respect to a range in which the optical system goes out of focus.
As described above, performing image recovery processing for a sensed input image can improve image quality by correcting aberrations.
Image recovery processing methods include a method of applying image recovery processing to a RAW image having a signal corresponding to one color component, namely one of the R, G, and B color components; and a method of applying image recovery processing to each color plane after performing interpolation so that each pixel has a signal corresponding to all color components, namely R, G, and B color components.
The method of applying image recovery processing to each color plane is larger than the method of applying image recovery processing to a RAW image in terms of the number of pixels to which image recovery processing is applied and the number of taps of the recovery filter. This leads to a considerable increase in the processing load of image recovery processing.
In general, the color components of the respective pixels constituting a RAW image are often arranged in a Bayer arrangement like that shown in FIG. 2. In this case, the number of pixels of a G component is larger than that of R or B components. For this reason, frequency characteristics in the pixel arrangement of a G component in a RAW image differ from those in the pixel arrangement of R and B components. As described above, since image recovery processing is equivalent to the correction of frequency characteristics, the frequency band of a G component differs from that of an R component or B component. In this case, a G component can be recovered in a higher frequency band than that of an R component or B component. If only a G component, out of R, G, and B components, is recovered up to a high frequency band, image recovery processing sometimes generates a false color which has not existed in the original image in an area that includes a high-frequency component in the image. This is because the relationship between the frequency characteristics of R, G, and B components in the high-frequency band of the image has changed before and after the image recovery processing. As described above, performing image recovery processing for signal components in different frequency bands will generate false colors. A false color in this case is generated due to a change in the pixel data itself, which is acquired by an image sensor, unlike a false color generated due to pixel interpolation for an image in a Bayer arrangement. Therefore, using a pixel interpolation algorithm designed to suppress the generation of false colors cannot suppress false colors generated via image recovery processing.